


Institutional Archive of the Naval Postgraduate School 





Calhoun: The NPS Institutional Archive 
DSpace Repository 


Theses and Dissertations 1. Thesis and Dissertation Collection, all items 


1971-03 


A comparison of the exact and approximate 
power of the chi-square goodness-of-fit test. 


Wright, Brian Theodore. 


Monterey, California ; Naval Postgraduate School 


http://ndl.handle.net/10945/15608 


Downloaded from NPS Archive: Calhoun 


Calhoun is the Naval Postgraduate School's public access digital repository for 
uh D U DLEY research materials and institutional publications created by the NPS community. 
«ili 3 Calhoun is named for Professor of Mathematics Guy K. Calhoun, NPS's first 
KN OX appointed — and published — scholarly author. 


INN LIBRARY Dudley Knox Library / Naval Postgraduate School 


411 Dyer Road / 1 University Circle 
Monterey, California USA 93943 








http://www.nps.edu/library 


A COMPARISON OF THE EXACT AND APPROXIMATE POWER 
OF THE CHI-SQUARE GOODNESS-OF-FIT TEST 


by 


Brian Theodore Wright 








p ere £ . ot . f 
Om, Ma f or ‘ Sait zoe. vipa — re mre . ork : ~ : : 5 2, pms, 5 ! 
ep ome ea ca >! as : a Pa ; , : Be [PR Oe oe | 
. £ ay ft , => 1. = ‘ . 2 ; f Pl ; a rt FF 

Nowe S208 » “ccs , eawt Vow “ne v “ = ‘our ‘ 409 #5 — Be Nene f8? Se . Nar DP hb fi Nea vce? xu 


eos <i -_ oe 
a ae oes ee 
:) — Ai ~p, a ; ‘ ff 


— a ee 
-_— em rp ree ag a rem ye rm RE Ae 


A COMPARISON OF THE EXACT AND APPROXIMATE POWFR 
Ome lie Cal-SQUARE GOODNESS-OF-FiIT TEST 


by 











ee SS Sw rete re sce 








| 
| 


| 
| 
Brian Theodore Wright 
Thesis Advisor: ese ce < 
March 1971 














2 hs i a eet SS 





A Comparison of the Fxact and Approximate Power 


of the Chi-soaguare Goodness-of-fit Test 


by 


Brian Theodore wright 
Captain, United States Marine Corps 
eeoe, Auburn University, 1966 


Submtcted in partial fulfillment of the 
requirements for the degree of 


Pee enReOr sSCIrPNCr iM OPFRATIONS RPSEARCH 


from the 


hoe eeeeOs EGRArUATC SCHOOL 
March 1971 





RY 





MTCRANVATE SCHOOL 


LK 
F rata 1 Yue 
, eb Ts 8 Oo 38 0) 


ARSTRACT' 


This thesis presents a numerical comparison of the exact 
and approximate powers of the chi-square goodness-of-fit 
test for small numbers of classes anc small sample sizes for 
the egquiprobahle null hypothesis. The comparison was per- 
formed using an IBM 360 computer and the computational 
See ls ale presented within the thesis. In addition a 
@emperisonm Of Critical points was conducted for the chi- 
Square distribution and tre associated exact, (multinomial), 
distribution. The results of the power comparisons show 
that the approximate power is surprisingly good and is 
recommenced as an efficient method for determining type two 
@rror associated with the test. Further, use of the chi- 
square distribution for determining a critical point is 
reinforced through the numerical comparison of significance 


levels. 
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lee INTRODUCTION 
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The power of the chi-square goodness-of-fit test has 
been an elusive problem which has attracted many authors, 
Prsenhart [1], Mann anc Wald [2], and Patnaik [3] have all 
presented expressions for approximating the power of the 
test for simple null hypotheses. hater work by Mitra [4] 
and Diamond [5] presented power functions for compound null 
hypotheses. These approximate power functions have all 
been developed through theoretical considerations, however 
it is not known how good they are for approximating the 
true power of the chi-square test. 

Cochran in his expository article [6] has presented a 
detailed history of the chi-square test. Included 1s a 
proposed method, (which he attributes to Tukey), for 
approximating the power of the chi-square test. This 
method has been referred to as the Pitman limiting power 
by a later author (cf. Mitra [4]). However the key idea 
appears to go hack to Eisenhart. 

Herein this approximation of power is compared with 
the true power as computed for the special case of the null 
hypothesis having egquiprobabhle classes and alternative 
hypotheses such that all classes but one are equiprohable. 
It is shown beet the approximation is reasonably good for 


small sample sizes. 





Mepiess ong been recognized that the chi-square test 
provides only an approximate critical region. Thus compari- 
Bens Of exact levels of significance with the approximate 
Bi@eeevere eisO in order. Such a comparison of significance. 
iowers and their associated critical points is presented. 

ee te © owt CeSection a discussion of the previous 
work performed in this area is presented along with notes 
On how this research fits into the scheme of the previous 
work. Section III presents the Fisenhart Beals approxima-~ 
tion of the power and is followed in Section IV hy the 
details of the special case used for the comparison of the 
exact and approximate power. The computational formulae 
for all the comparisons appear in Section V and the results 


Meemeconuclusitons are discussed in Section VI. 
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II. DISCUSSION AMD TYE NATURP OF THF PROBLEM 


we ee 


In their 1931 paper Neyman and Pearson [7] presented an 
example of a three class multinomial probability function 
with a sample size of 10 observations. They observed mee 
the probability calculations from the chi-square dcistribu- 
tion were on a whole better than expected. Wowever they 
opened to question the use of the chi-square approximation 
when the class expectations are small with respect to the 
sample size. This question has been answered only with 
hueristic suggestions in the literature, (cf. Cochran [8] 
and Watson [9]). The research reported within this paper 
sheds some additional light on this question. 

Hoel in his 1938 paper [10] pointed out that there are 
two types of error associated with the chi-square goodness-~ 
of-fit test for small sample sizes. The first type of 
Srror arose from the fact that the derivation of the test 
criterion was based on rough approximations. Whereas the 
second type of error arose from using an integral of a con- 
tinuous function instead of summing the appropriate terms 
of a discrete distribution to determine significance levels. 
Hoel concludes in his paper that errors based upon the 
deeivarigiwer tie Criterion from rough approximations are 
not significant. However he leaves untouched any discussion 
Sieiow Signikicant are the errors obtained by using an 


integral of a continuous function instead of summing the 





Giscrete terms. Further, no research was uncovered which 
fully answered Hoel‘'s question. This paper will help 
provide an answer to that question. 

The majority of the work on the power function conducted 
meets £t1eC10d since 1945 has heen concerned with theoretical 
developments using compound hypotheses. However Watson [9] 
in an expository article on recent results points out that 
the test has still not had any computations made regarding 
its power for small sample sizes, and he suggested some 
means of electronic calculation be verformed to evaluate the 
power of the test. This in summary is what this paper 


MEeSentcs. 





TIT. APPROXIMATION TO POWFR 
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The work reported in this section was proposed hy 
several authors, however Fisenhart [1] is believed to he 
the first author to present the method. Hence the approx- 
imation to power is hereafter referred to as the approx- 
miettOn Cue to Fisenhart et al 

Pee mOwniethat the chi-square goodness-of-fit test 
iS a cons:zstent test. As the number of ohservations taken 
from the sampled distribution incr2ases, then the power of 
Bie test tends to unity for all alternative hypotheses. 
Thus the family of power curves might look like those in 
Figure 1 below, where W represents the distribution 
Specified by the null hypothesis. 

Tieteerealiveene as  Pithe test statistic > critical 
point|w} +1.0 as N+” for each we{H,}, the set of alterna- 
tive hypotheses. In order to make this limit less than 
unity, it is necessary to choose a sequence of alternative 


hypotheses w,, converging to W, as N>o. The sequence 


N 
Bay (Wy) ZeEeieomsest statistic > critical point |w,,} might 
converge to some value § <1.0 as indicated by the asterisks 


in Figure 1. By appropriate choice of the sequence Ong 
the corresponding probabilities representing the power, 
Bag Wag) will converge to £B rapidly (become and remain close 


to 6 for rather small N). Thus the power for finite N can 


be approximated by 8. On the other hand, an inappropriately 
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Members of the set of alternative hypotheses 


Figure 1. A Schematic Representation of Pisenhart, 
eee one OG OF Approximating the Power. 


t 
chosen sequence a would not converge rapiclv to f and hence 


, 
pe Ould be aA poor approximation to By, (a7) for many finite 


values of N. Such a sequence is indicated with dots in 


Figure 1. Thus the choice of sequences Or er Ctl eae 


The following expressions are presented for approximat- 


ing the power (1i.e., choosing the sequence) of the simple 


emi-sQuare Goodness-of-fit test. 


Let the null hypothesis ie Cescribe Kk class prohabilities 
Pyre++rP,, and let the alternative hypotheses a he 
described hy different choices of class probabilities, 


e.g., all possible ee: different from Ae: Define a 


Bem; = 1p... ,% by as =p, + 0,/vN where N is the 


Muniset Or OosSservations . Tavs for a fixed alternative 


ree Da: and fixed MN, the null hypothesis and the 





alternative are connected by the tG.)}. As N > © then the on 


serve as the sequence w,, and conve:ge to Pp; which serve 


N 
as W.. 
k k k 
ieee merece that © p. =~ l= ££ p mence <Le-O.. 20 
i i 
: i=] L=L 1=1 
ini Goran 
and 0. yN (D, p.)- 
Let x, 1 =1,...,k Cescribe the ohserved frequency with 


which observations fall into frecuency class i, then define 
a new term q; as the observed portion of observations 
fming into class i, i.e. a. = x. /N. 


Wrmeme@eat StotlStilc can therefore ba defined to be 
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with all cross product terms reduced to zero due to the 


k 
Mmeeerrction that 2 9. = 4. 
1=1 
leeveom@ced by Cochran [6], that the test statistic 


then has a non-central chi-square distribution (in the 


Piiemaem Nes 9) With non-centrality parameter 


Lin: 
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IV. DETAILS OF THF SPPCIAL CASF 








AS a sequence of alternative hypotheses had heen proposed 
which converge to the null hypothesis, the following special 
case was developed to compare the approximation with the 
exact power of the chi-square test. 

As Mann and Wald [2] pointed out in their paper, every 
Semetimuous probability distribution can be transformed .into 
met Orm GQiStribution on the interval (0,1). Therefore 
the null hypothesis for the special case was that the classes 
of the muitinomial were chosen such that they were described 
by equal class probabilities, i.e. eo Dee lakes bps res oy 
where k is the number of classes. 

The only alternative hypotheses considered were those 
that specify equal probabilities for all classes hut one. 
Sance : Oy = 1, these alternative hypotheses may be 
| poetollewse. het a ei de he = A eam 


Py. = 1 - (k-1)p/k. Then the non-centrality parameter was 


computed from 


a ; 
er ee = N (k-1) (=) 
j=1 Pi 


Diiemealues Or 6 USec herein were .2, .5, .8. 
The sequence of alternative hypotheses used for the 
comparison of the approximation was chosen due to its sim- 


plicity and rather extreme character; ail but one cell being 


ie 





equiprobahle and the one cell having a surplus of probh- 
ability. The same scheme with 9 > ] would he less extrane, 
Obviously, the same value for the non-centrality parameter 
can he realized with other alternative hypotheses. It is 
proposed to research the question of whether or not the 
peeeeentiy used scheme has a sense of extremity in terms of 


power when the non-centrality parameter is held fixed, 
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The prohlem was to co:ympute the prohahility that the sum 
of squares of the class frequencics exceeded a predefined 
critical point. The work Gone previously in this area was 
Mainly pexformed during the 1930's. At that time the size 
of the task was enormous and aliost impossible since the 
researchers did not have the aid of modern electronic 
computation eoauipment. 

Tt was decided that in order to gain enough information 
to have a meaningful presentation some type of computer 
program was required. The task involved writing a program 
which would generate the class partitions of the multinomial 
distribution and then compute tre sum of squares for each of 
the generated partitions. Once the sum of squares had been 
computed the sum was checked to ensure that it was greater 
miameecte Critical point as specified from the chi~square 
distribution. If the sum of squares did exceed the critical 
point then the multinomial probability associated with the 
partition was computed. If the sum was less than or equal 
Peete memmalcal point that partition was ignored and the 
program generated another partition and the process was 
repeated, 

The multinomial probabilities associated with a fixed 
k part partition of N must be summed over all permutations 


eee War titton. Thus 


rs 








geen pee iz 
> See! eo Py L 
qee | IL k 
where K. Teen mMrivCr OF Ways xX. ,..2e7R. x, are cae 
1 ar ee ee” apis 
could occur in the (k-1) remaining classes. The computations 


were perrormed in this manner due to the nature of the 
alternative hypotheses, with the first (k-1) classes of 
equal probability ey and the oo class of prohability 
Bee (1 = (k-1)ps). 

One of the inefficiencies of the program was that it 
must compute Key the occurrence coefficient, k times for 
each of the generated partitions. This calculation required 
the computer to search through each generated partition and 
compute the number of occurrences of each possible Xe within 
the partition, and then determine the appropriate 
SCembinatoric. 

To give an example of the foregoing, consider the follov- 
meg paktition for a 6 class multinomial distrihution with 
twelve @reeteaetous, (5,3,3,1;0,0). The multinomial 


ee 


coefficient is BTRIaITT OF Geo e ee The partition prohability 


under the alternative hypothesis is therefore 


faa oN Jens iliak A 
oO! oO O-_ Oo O O O 
665,280 (Ds Py Ky eee Ky eee Py De Ky) where KF, 


Mie oPomateo the associatrea occurrence coefficients for 


the x. 'S. For the partition under consideration the K's 
a a: 


imme ee oto gn 4 trata: = 8°: «= S 8 
check it 1s noted that the total number of ways the partition 
Comlcameceur in a Six class law is peri eteeet SPO dic alice 

A Ne ee eis 

y ge aes Oi 
i1=1 
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Having computed the probahilitv under the alternative 
MemeOucNesis thay the test statistic is greater than the chi- 
Square critical point, the non-centrality parameter lambda 
was compuzed for the k classes, the N ohservations and the 
appropriate 9. With this parameter and (k-1) degrees of 
freedom the approximate power was computed utilizing the 
Mem—-Ccentral chi-square distribution. 

The computation of the approximate power was performed 
Memg a method found in Fix [11]. The power function of 


Peiewchii-Ssquare goodness-of-fit test heing approximated hy 


meee U2)! x tein 4 “ix? 
3=0 ‘ x2 ae P45) 


Ce ac) 


where (k+2j-2) are the dearees of freedom for the incomplete 


2 ae . 
eammea Cunction and % tS tii Critical point of the 


k-1) (a) 
chi-square with (k-1) degrees of freedom and the specified 
alpha level. The evaluation of the incomplete gamma function 
was accomplished by using a previously prepared program by 
Monn RR. B. Whittlesey of UCLA, a listing of his program 
appears in the computer listing section of this paper. 

Aside from the method employed to compute these prob- 
abilities there was nothing new in the theory employed. [In 
fact this theory has been and continues to be the standard 
method of evaluating the power. The method used to generate 
the class Ree Sons was Original and was the important step 


Mmeciemosecess Of Allowing large amounts of data to be col- 


lected in a relatively small amount of time. The ordering 


is 





Of the sum of squares of the k-pare partitions of BF is an 
important integer programming problem. Problems of this 
Pype are Ciscussed in a survey article hy Saaty [12]. 

since the sum of squares of the generated partitions 
yields an integer value, the folloving method was used to 
Seeeulate the critical point for the chi-square, and the 
exact Critical points as determined under the equiprohable 
null hypothesis. 

Let C, be the critical point read from the chi-square 
table for (k-1) degrees of freedom (this is the value cor- 
jMespondang to a k class multinomial distribution). The 
Meonpabilaty that the test statistic exceeds this critical 
meame 15 alpha, hence the following is true. If the test 


Seeacastic is 


kk (x, -Npi) 
a Npi 
fei) P 
and 
ei xe = Ba) ‘ 
P Y j}———______ > ae 
sil er + “e 


as specified by the test, then 


k 
g N 
Pees = (C-+N) == a 
i=l 7 alae 
since on i Wie tnen since Lie sum of the squares of inte- 


Gee veomogain an integer the critical point or interest is 


the greatest integer in [(C. +N) Roi 


LEG 





Me exact Critical points mma deterpinec) from the equi- 
Probable multinomial by considering in turn each value for 
the sum of squares in decreasing order. Until a value was 
Peachea which yielded a probability slightly greater than 
ne alpha wrohability. Then the process was repeated 
With the next smallest value such that the alpha level was 
bracketed. These two values hecame the upper critical 


point CR and the lower critical point CR respectively. 
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The results of the comparison of the approximate anc 
exact powers of the chi-souare test are found in Tables 2 
through 17 in the computer output section of this thesis. 
These tables present the data in the following manner. For 
each value of p and alpha considered there are five columns; 
ee esos Che Numoer Of classes k, the second the 
number of observations N, the thirds the exact power as 
computed from the associated k class multinomial distrihu- 
tion, the fourth displays the approximate power as computec 
wemeoemethnod £Ound in Fax [il], and the fifth column shows 
the associated non-centrality parameter for the approximation. 

iieeoreet =O PLOoviace a more concise display of informa- 
tion, four graphs, Figures 3 through 6, have heen prepared 
which correspond to the data found in Tables 2 through 5. 
Several conclusions were drawn regarding the data found in 
tne tables and the four graphs. 

Dioula was 1OEGd that as the Geviation, (l-o), 
increased hetween the null and alternative hypotheses the 
power of the test increased very rapidly with MN. Secondly 
it was noted that the approximation of power was generally 
more conservative when the deviation between hypotheses was 
small, and that the approximation was generally over opti- 


firsivenron Jarge N and (1 - p). 





However it should be noted that as am approximation Lhe 
asymptotic power is quite cood especially as a means for 
determining how large a sample size is required to yield 
a specific level of power. Further, use of the non-central 
chi-square for estimating the probability of type II error 
associated with the test shoule he encouraged since it is en 
efficient method amenable to all alternative hypotheses. 

The results of the comparison of significance levels 
Meme me seee Critical points and their associated alpha 
levels are presented in Table 1. The data presentec are fcr 
egquiprobable multinomial distributions of three, four and 
five classes. Table 1 presents the data in the following 
Meier. There are three divisions in the table, the first 
is a reference division showing class size and the number 
of observations, the second division is for the data 
Meeectated With an alpha of .05, the third division is for 
the data associated with an alpha of .01. Within each of 
the latter two divisions there appear five columns; the 
First of these displays the greatest integer in the critical 
point calculation from the chi-square table (see section on 
computational details for further explanation), the second 
column presents the lower exact critical point, the third 
column presents the significance level associated with this 
Satene eponier im a like manner the fourth and fifth columns 
present the same data for the upper critical point. 

Figure 2 presents a graphical representation of the data 


Eomiicmiime la lem letor a) tour class multinomial distribution. 


Ie, 
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Figure 2. Significance Levels of Calculated Critical POLNtS. 
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The upper and lower critical points straddle their re®@pec-: 
tive alpha level and are connected bv a straight line. In 
the case where the two alpha levels share common upper and 
lower critical points a double line connects the two asso- 
ciated Shand ticance levels. 

The data vresented in Table 1 and the graph, Figure 2, 
andicate that the critical point associated with the chi- 
square test 1S a very good approximation to the exact 
oeeeetea | | pOints, and is always bounded by the upper critical 
point. In those cases where it is not bounded by the lower 
critical point it does increase the probability of type I 


error, however this occurs infrequently. 


23 





Test 


the 
Ut 
Ca 
1 





Power of 





- = 
SP a a ER a TS NA Sa a. 
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Mumaer of Observations N 


Pegitemeocn Ne COMPDartson Of Ehe Fract anc Asymptotic 
Powers of the Chi-square Test for a Three Class 
Multinomial Distribution with an Alpha = .05. 

The following remarks are applicable to Figures 3, 4, 5 
and 6 only. The asymptotic or approximate power appears as 
the smooth curves in all figures, whereas the exact power 
curves are the jagged lines connecting the heavy dots. The 
Oo values indicated to the left of the curves were those 
associated with on = p/k for the alternative hypotheses. 
The power of the test is plotted on the ordinate versus the 


number of observations, MN, as plotted on the abcissa. 


This figure corresponds to the data presented in Table 2. 
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Figure 4. A Comparison of the Fxact and Asymptotic 
POweeemoteene Cia=-square Test for a Three Class 
ire lnomeal Drstribution with an Alpha = .01. 


Seommeoees EOr Figure 3. 


This figure corresponds to the data presented in Table 3. 
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Powers of the Chi-square Test for a Four Class 
Multinomial Distribution with an Alpha = .05. 


See notes for Figure 3. 


This figure corresponds to the data presented in Table 4. 
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Figure 6. A Comparison of the Fact and Asymptotic 


Powers of the Chi-square Test for a Four Class 
ition Stp lave rom with an Alpha = .01. 


See notes for Figure 3. 


This figure corresponds to the data presented in Tahle 5. 
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